{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "yADFA_tU_fmz"
   },
   "source": [
    "# Introduction to Vertex Pipelines\n",
    "\n",
    "\n",
    "\n",
    "## Learning Objectives\n",
    "\n",
    "1. Use the Kubeflow Pipelines SDK to build scalable ML pipelines.\n",
    "\n",
    "2. Create and run a 3-step intro pipeline that takes text input.\n",
    "\n",
    "\n",
    "\n",
    "## Introduction \n",
    "\n",
    "In this notebook, you will learn how to create and run ML pipelines with Vertex Pipelines.\n",
    "\n",
    "\n",
    "Each learning objective will correspond to a __#TODO__ in the [student lab notebook](../labs/intro_to_vertex_pipelines.ipynb) -- try to complete that notebook first before reviewing this solution notebook.\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "rOoM9J7S1s11"
   },
   "source": [
    "# Vertex Pipelines setup\n",
    "\n",
    "There are a few additional libraries we'll need to install in order to use Vertex Pipelines:\n",
    "\n",
    "* **Kubeflow Pipelines**: This is the SDK we'll be using to build our pipeline. Vertex Pipelines supports running pipelines built with both Kubeflow Pipelines or TFX.\n",
    "\n",
    "* **Google Cloud Pipeline Components**: This library provides pre-built components that make it easier to interact with Vertex AI services from your pipeline steps.\n",
    "\n",
    "\n",
    "### Step 1: Install necessary libraries\n",
    "\n",
    "To install the setuptools package run the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture --no-stderr\n",
    "!pip install setuptools==59.5.0"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To install both services we'll be using in this lab, first set the user flag in a notebook cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "fuMBD_m32SOM"
   },
   "outputs": [],
   "source": [
    "USER_FLAG = \"--user\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "vPMR5FHA2Ubk"
   },
   "source": [
    "Then run the following from your notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "NGGHQHdq2WWv",
    "outputId": "174d080b-6245-464b-f7b3-9981e0282dca"
   },
   "outputs": [],
   "source": [
    "%%capture --no-stderr\n",
    "# Install necessary libraries\n",
    "!pip3 install {USER_FLAG} google-cloud-aiplatform==1.0.0 --upgrade\n",
    "!pip3 install {USER_FLAG} kfp google-cloud-pipeline-components==0.1.1 --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture --no-stderr\n",
    "!pip install -U google-cloud-aiplatform \"shapely<2\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please ignore any compatibility warnings and errors.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "SSTpQUrt2XYC"
   },
   "source": [
    "After installing these packages you'll need to restart the kernel:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "V9irSOLF2caE"
   },
   "outputs": [],
   "source": [
    "# Load necessary libraries\n",
    "import os\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    # Automatically restart kernel after installs\n",
    "    import IPython\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "KpyDK82U2dLM"
   },
   "source": [
    "Finally, check that you have correctly installed the packages. **The KFP SDK version should be >=1.6**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "GGFFardW2iGM",
    "outputId": "f2d61fbf-ef4b-4a28-f38c-b7e98d520e8d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KFP SDK version: 1.8.11\n",
      "google_cloud_pipeline_components version: 0.1.1\n"
     ]
    }
   ],
   "source": [
    "# print KFP SDK version\n",
    "!python3 -c \"import kfp; print('KFP SDK version: {}'.format(kfp.__version__))\"\n",
    "!python3 -c \"import google_cloud_pipeline_components; print('google_cloud_pipeline_components version: {}'.format(google_cloud_pipeline_components.__version__))\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "WfjVrXSz2i7W"
   },
   "source": [
    "### Step 2: Set your project ID and bucket\n",
    "\n",
    "Throughout this lab you'll reference your Cloud project ID and the bucket you created earlier. Next we'll create variables for each of those.\n",
    "\n",
    "If you don't know your project ID you may be able to get it by running the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pNEfOW8H2olJ",
    "outputId": "5a51fa9e-ae88-40b3-ffa2-504001d115b7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Project ID:  cloud-training-prod-bucket\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "PROJECT_ID = \"\"\n",
    "# Get your Google Cloud project ID from gcloud\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    shell_output=!gcloud config list --format 'value(core.project)' 2>/dev/null\n",
    "    PROJECT_ID = shell_output[0]\n",
    "    print(\"Project ID: \", PROJECT_ID)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "BCvvh2wn2rVb"
   },
   "source": [
    "Otherwise, set it here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6HY1T8ga2srn"
   },
   "outputs": [],
   "source": [
    "if PROJECT_ID == \"\" or PROJECT_ID is None:\n",
    "    PROJECT_ID = \"your-project-id\"  # TODO: Replace with your actual Google Cloud Project ID"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "r3rkCTRV2uMw"
   },
   "source": [
    "Then create a variable to store your bucket name. If you created it in this lab, the following will work. Otherwise, you'll need to set this manually:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "iAMz8MIS2zG4"
   },
   "outputs": [],
   "source": [
    "BUCKET_NAME=\"gs://\" + PROJECT_ID + \"-bucket\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "QWLzFXJS20eV"
   },
   "source": [
    "### Step 3: Import libraries\n",
    "\n",
    "Add the following to import the *libraries* we'll be using throughout this lab:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "ZzNbMa2923ue"
   },
   "outputs": [],
   "source": [
    "# Load necessary libraries\n",
    "from typing import NamedTuple\n",
    "import kfp\n",
    "from kfp import dsl\n",
    "from kfp.v2 import compiler\n",
    "from kfp.v2.dsl import (Artifact, Dataset, Input, InputPath, Model, Output,\n",
    "                        OutputPath, ClassificationMetrics, Metrics, component)\n",
    "from kfp.v2.google.client import AIPlatformClient\n",
    "from google.cloud import aiplatform\n",
    "from google_cloud_pipeline_components import aiplatform as gcc_aip"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "Z36Z9iyY25ca"
   },
   "source": [
    "### Step 4: Define constants\n",
    "\n",
    "The last thing we need to do before building our pipeline is define some constant variables. PIPELINE_ROOT is the Cloud Storage path where the artifacts created by our pipeline will be written. We're using us-central1 as the region here, but if you used a different region when you created your bucket, update the REGION variable in the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 72
    },
    "id": "wl6_Tncc2-eF",
    "outputId": "0fef9f10-593d-4dcc-a3ca-55655ed5263b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: PATH=/usr/local/cuda/bin:/opt/conda/bin:/opt/conda/condabin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/jupyter/.local/bin\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'gs://cloud-training-prod-bucket-bucket/pipeline_root/'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "PATH=%env PATH\n",
    "%env PATH={PATH}:/home/jupyter/.local/bin\n",
    "REGION=\"us-central1\"\n",
    "PIPELINE_ROOT = f\"{BUCKET_NAME}/pipeline_root/\"\n",
    "PIPELINE_ROOT"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "Gz4Fg27Q2_KH"
   },
   "source": [
    "After running the code above, you should see the root directory for your pipeline printed. This is the Cloud Storage location where the artifacts from your pipeline will be written. It will be in the format of gs://Your_Project_ID/pipeline_root/"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "fqiZ3IFC3EkX"
   },
   "source": [
    "# Creating your first pipeline\n",
    "\n",
    "To get familiar with how Vertex Pipelines works, we'll first create a short pipeline using the KFP SDK. This pipeline doesn't do anything ML related (don't worry, we'll get there!), we're using it to teach you:\n",
    "\n",
    "* How to create custom components in the KFP SDK\n",
    "* How to run and monitor a pipeline in Vertex Pipelines\n",
    "\n",
    "We'll create a pipeline that prints out a sentence using two outputs: a product name and an emoji description. This pipeline will consist of three components:\n",
    "\n",
    "* **product_name**: This component will take a product name (or any noun you want really) as input, and return that string as output\n",
    "* **emoji**: This component will take the text description of an emoji and convert it to an emoji. For example, the text code for ✨ is \"sparkles\". This component uses an emoji library to show you how to manage external dependencies in your pipeline\n",
    "* **build_sentence**: This final component will consume the output of the previous two to build a sentence that uses the emoji. For example, the resulting output might be \"Vertex Pipelines is ✨\".\n",
    "\n",
    "Let's start coding!\n",
    "\n",
    "### Step 1: Create a Python function based component\n",
    "\n",
    "Using the KFP SDK, we can create components based on Python functions. We'll use that for the 3 components in our first pipeline. We'll first build the **product_name** component, which simply takes a string as input and returns that string. Add the following to your notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "OlBup0Hk3XA1"
   },
   "outputs": [],
   "source": [
    "@component(base_image=\"python:3.9\", output_component_file=\"first-component.yaml\")\n",
    "def product_name(text: str) -> str:\n",
    "    return text"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "ZPv2KpJB3eDo"
   },
   "source": [
    "Let's take a closer look at the syntax here:\n",
    "\n",
    "* The @component decorator compiles this function to a component when the pipeline is run. You'll use this anytime you write a custom component.\n",
    "\n",
    "* The base_image parameter specifies the container image this component will use.\n",
    "\n",
    "* The output_component_file parameter is optional, and specifies the yaml file to write the compiled component to. After running the cell you should see that file written to your notebook instance. If you wanted to share this component with someone, you could send them the generated yaml file and have them load it with the following:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "VywgRJgQ3kAa"
   },
   "outputs": [],
   "source": [
    "# TODO\n",
    "product_name_component = kfp.components.load_component_from_file('./first-component.yaml')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "zLyxQN3G3llo"
   },
   "source": [
    "* The -> str after the function definition specifies the output type for this component."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "xqmeRU9Y3qmm"
   },
   "source": [
    "### Step 2: Create two additional components\n",
    "\n",
    "To complete our pipeline, we'll create two more components. The first one we'll define takes a string as input, and converts this string to its corresponding emoji if there is one. It returns a tuple with the input text passed, and the resulting [emoji](https://pypi.org/project/emoji/):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "R6xzpY5d3w5C"
   },
   "outputs": [],
   "source": [
    "@component(packages_to_install=[\"emoji==1.6.3\"])\n",
    "def emoji(\n",
    "    text: str,\n",
    ") -> NamedTuple(\n",
    "    \"Outputs\",\n",
    "    [\n",
    "        (\"emoji_text\", str),  # Return parameters\n",
    "        (\"emoji\", str),\n",
    "    ],\n",
    "):\n",
    "    import emoji\n",
    "    emoji_text = text\n",
    "    emoji_str = emoji.emojize(':' + emoji_text + ':', use_aliases=True)\n",
    "    print(\"output one: {}; output_two: {}\".format(emoji_text, emoji_str))\n",
    "    return (emoji_text, emoji_str)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "OkM9g3jA30wA"
   },
   "source": [
    "This component is a bit more complex than our previous one. Let's break down what's new:\n",
    "\n",
    "* The packages_to_install parameter tells the component any external library dependencies for this container. In this case, we're using a library called emoji.\n",
    "* This component returns a NamedTuple called Outputs. Notice that each of the strings in this tuple have keys: emoji_text and emoji. We'll use these in our next component to access the output.\n",
    "\n",
    "The final component in this pipeline will consume the output of the first two and combine them to return a string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "7i7aTotY36rk"
   },
   "outputs": [],
   "source": [
    "@component\n",
    "def build_sentence(\n",
    "    product: str,\n",
    "    emoji: str,\n",
    "    emojitext: str\n",
    ") -> str:\n",
    "    print(\"We completed the pipeline, hooray!\")\n",
    "    end_str = product + \" is \"\n",
    "    if len(emoji) > 0:\n",
    "        end_str += emoji\n",
    "    else:\n",
    "        end_str += emojitext\n",
    "    return(end_str)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "5x2hI02j3-yq"
   },
   "source": [
    "You might be wondering: how does this component know to use the output from the previous steps you defined? Good question! We will tie it all together in the next step.\n",
    "\n",
    "### Step 3: Putting the components together into a pipeline\n",
    "\n",
    "The component definitions we defined above created factory functions that can be used in a pipeline definition to create steps. To set up a pipeline, use the @dsl.pipeline decorator, give the pipeline a name and description, and provide the root path where your pipeline's artifacts should be written. By artifacts, we mean any output files generated by your pipeline. This intro pipeline doesn't generate any, but our next pipeline will.\n",
    "\n",
    "In the next block of code we define an intro_pipeline function. This is where we specify the inputs to our initial pipeline steps, and how steps connect to each other:\n",
    "\n",
    "* product_task takes a product name as input. Here we're passing \"Vertex Pipelines\" but you can change this to whatever you'd like.\n",
    "\n",
    "* emoji_task takes the text code for an emoji as input. You can also change this to whatever you'd like. For example, \"party_face\" refers to the 🥳 emoji. Note that since both this and the product_task component don't have any steps that feed input into them, we manually specify the input for these when we define our pipeline.\n",
    "\n",
    "* The last step in our pipeline - consumer_task has three input parameters:\n",
    "\n",
    "* The output of product_task. Since this step only produces one output, we can reference it via product_task.output.\n",
    "\n",
    "* The emoji output of our emoji_task step. See the emoji component defined above where we named the output parameters.\n",
    "\n",
    "* Similarly, the emoji_text named output from the emoji component. In case our pipeline is passed text that doesn't correspond with an emoji, it'll use this text to construct a sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "v0_LaqHl4Ksj"
   },
   "outputs": [],
   "source": [
    "@dsl.pipeline(\n",
    "    name=\"hello-world\",\n",
    "    description=\"An intro pipeline\",\n",
    "    pipeline_root=PIPELINE_ROOT,\n",
    ")\n",
    "# You can change the `text` and `emoji_str` parameters here to update the pipeline output\n",
    "def intro_pipeline(text: str = \"Vertex Pipelines\", emoji_str: str = \"sparkles\"):\n",
    "    product_task = product_name(text)\n",
    "    emoji_task = emoji(emoji_str)\n",
    "    consumer_task = build_sentence(\n",
    "        product_task.output,\n",
    "        emoji_task.outputs[\"emoji\"],\n",
    "        emoji_task.outputs[\"emoji_text\"],\n",
    "    )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "j4CXV_Vt4Mwy"
   },
   "source": [
    "### Step 4: Compile and run the pipeline\n",
    "\n",
    "With your pipeline defined, you're ready to compile it. The following will generate a JSON file that you'll use to run the pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "fR19GLwd4Rrw"
   },
   "outputs": [],
   "source": [
    "compiler.Compiler().compile(\n",
    "    pipeline_func=intro_pipeline, package_path=\"intro_pipeline_job.json\"\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "DbMOA3yK4Tne"
   },
   "source": [
    "Next, instantiate an API client:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "hU-jMJ4n4Zuy"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jupyter/.local/lib/python3.7/site-packages/kfp/v2/google/client/client.py:173: FutureWarning: AIPlatformClient will be deprecated in v2.0.0. Please use PipelineJob https://googleapis.dev/python/aiplatform/latest/_modules/google/cloud/aiplatform/pipeline_jobs.html in Vertex SDK. Install the SDK using \"pip install google-cloud-aiplatform\"\n",
      "  category=FutureWarning,\n"
     ]
    }
   ],
   "source": [
    "api_client = AIPlatformClient(\n",
    "    project_id=PROJECT_ID,\n",
    "    region=REGION,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "5gBEgxF84bqi"
   },
   "source": [
    "Finally, run the pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "S2qkfsmN4d_r"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "See the Pipeline job <a href=\"https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/hello-world-20220227080911?project=cloud-training-prod-bucket\" target=\"_blank\" >here</a>."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# TODO\n",
    "response = api_client.create_run_from_job_spec(\n",
    "    job_spec_path=\"intro_pipeline_job.json\",\n",
    "    # pipeline_root=PIPELINE_ROOT  # this argument is necessary if you did not specify PIPELINE_ROOT as part of the pipeline definition.\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "RvSIjL4G4gkp"
   },
   "source": [
    "Running the pipeline should generate a link to view the pipeline run in your console. It should look like this when complete:\n",
    "\n",
    "![sr1.png]()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "8oYBWf4w4vED"
   },
   "source": [
    "This pipeline will take **5-6 minutes** to run. When complete, you can click on the build-sentence component to see the final output:\n",
    "\n",
    "![build-sentence.png]()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "OU5SqYFG422X"
   },
   "source": [
    "Now that you're familiar with how the KFP SDK and Vertex Pipelines works, you're ready to build a pipeline that creates and deploys an ML model using other Vertex AI services. Be sure to do try the next lab, Create and run ML pipelines with Vertex Pipelines! "
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "intro-to-vertex-pipelines.ipynb",
   "provenance": []
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-6.m89",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-6:m89"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
